A merit function approach for evolution strategies

In this paper, we extend a class of globally convergent evolution strategies to handle general constrained optimization problems. The proposed framework handles quantifiable relaxable constraints using a merit function approach combined with a specific restoration procedure. The unrelaxable constraints, when present, can be treated either by using the extreme barrier function or through a projection approach. Under reasonable assumptions, the introduced extension guarantees to the regarded class of evolution strategies global convergence properties for first order stationary constraints. Numerical experiments are carried out on a set of problems from the CUTEst collection as well as on known global optimization problems

Globally convergent evolution strategies with application to Earth imaging problem in geophysics

Au cours des dernières années, s’est développé un intérêt tout particulier pour l’optimisation sans dérivée. Ce domaine de recherche se divise en deux catégories: une déterministe et l’autre stochastique. Bien qu’il s’agisse du même domaine, peu de liens ont déjà été établis entre ces deux branches. Cette thèse a pour objectif de combler cette lacune, en montrant comment les techniques issues de l’optimisation déterministe peuvent améliorer la performance des stratégies évolutionnaires, qui font partie des meilleures méthodes en optimisation stochastique. Sous certaines hypothèses, les modifications réalisées assurent une forme de convergence globale, c’est-à-dire une convergence vers un point stationnaire de premier ordre indépendamment du point de départ choisi. On propose ensuite d’adapter notre algorithme afin qu’il puisse traiter des problèmes avec des contraintes générales. On montrera également comment améliorer les performances numériques des stratégies évolutionnaires en incorporant un pas de recherche au début de chaque itération, dans laquelle on construira alors un modèle quadratique utilisant les points où la fonction coût a déjà été évaluée. Grâce aux récents progrès techniques dans le domaine du calcul parallèle, et à la nature parallélisable des stratégies évolutionnaires, on propose d’appliquer notre algorithme pour résoudre un problème inverse d’imagerie sismique. Les résultats obtenus ont permis d’améliorer la résolution de ce problème. ABSTRACT : In recent years, there has been significant and growing interest in Derivative-Free Optimization (DFO). This field can be divided into two categories: deterministic and stochastic. Despite addressing the same problem domain, only few interactions between the two DFO categories were established in the existing literature. In this thesis, we attempt to bridge this gap by showing how ideas from deterministic DFO can improve the efficiency and the rigorousness of one of the most successful class of stochastic algorithms, known as Evolution Strategies (ES’s). We propose to equip a class of ES’s with known techniques from deterministic DFO. The modified ES’s achieve rigorously a form of global convergence under reasonable assumptions. By global convergence, we mean convergence to first-order stationary points independently of the starting point. The modified ES’s are extended to handle general constrained optimization problems. Furthermore, we show how to significantly improve the numerical performance of ES’s by incorporating a search step at the beginning of each iteration. In this step, we build a quadratic model using the points where the objective function has been previously evaluated. Motivated by the recent growth of high performance computing resources and the parallel nature of ES’s, an application of our modified ES’s to Earth imaging Geophysics problem is proposed. The obtained results provide a great improvement for the problem resolution

Convergence and Complexity Analysis of a Levenberg–Marquardt Algorithm for Inverse Problems

The Levenberg–Marquardt algorithm is one of the most popular algorithms for finding the solution of nonlinear least squares problems. Across different modified variations of the basic procedure, the algorithm enjoys global convergence, a competitive worst-case iteration complexity rate, and a guaranteed rate of local convergence for both zero and nonzero small residual problems, under suitable assumptions. We introduce a novel Levenberg-Marquardt method that matches, simultaneously, the state of the art in all of these convergence properties with a single seamless algorithm. Numerical experiments confirm the theoretical behavior of our proposed algorithm

Inexact Direct-Search Methods for Bilevel Optimization Problems

In this work, we introduce new direct search schemes for the solution of bilevel optimization (BO) problems. Our methods rely on a fixed accuracy black box oracle for the lower-level problem, and deal both with smooth and potentially nonsmooth true objectives. We thus analyze for the first time in the literature direct search schemes in these settings, giving convergence guarantees to approximate stationary points, as well as complexity bounds in the smooth case. We also propose the first adaptation of mesh adaptive direct search schemes for BO. Some preliminary numerical results on a standard set of bilevel optimization problems show the effectiveness of our new approaches

A Subsampling Line-Search Method with Second-Order Results

In many contemporary optimization problems such as those arising in machine learning, it can be computationally challenging or even infeasible to evaluate an entire function or its derivatives. This motivates the use of stochastic algorithms that sample problem data, which can jeopardize the guarantees obtained through classical globalization techniques in optimization such as a trust region or a line search. Using subsampled function values is particularly challenging for the latter strategy, which relies upon multiple evaluations. On top of that all, there has been an increasing interest for nonconvex formulations of data-related problems, such as training deep learning models. For such instances, one aims at developing methods that converge to second-order stationary points quickly, i.e., escape saddle points efficiently. This is particularly delicate to ensure when one only accesses subsampled approximations of the objective and its derivatives. In this paper, we describe a stochastic algorithm based on negative curvature and Newton-type directions that are computed for a subsampling model of the objective. A line-search technique is used to enforce suitable decrease for this model, and for a sufficiently large sample, a similar amount of reduction holds for the true objective. By using probabilistic reasoning, we can then obtain worst-case complexity guarantees for our framework, leading us to discuss appropriate notions of stationarity in a subsampling context. Our analysis encompasses the deterministic regime, and allows us to identify sampling requirements for second-order line-search paradigms. As we illustrate through real data experiments, these worst-case estimates need not be satisfied for our method to be competitive with first-order strategies in practice

A Parallel Evolution Strategy for Acoustic Full-Waveform Inversion

In this work, we propose another alternative to find an initial velocity model for the acoustic FWI without any physical knowledge. Motivated by the recent growth of high performance computing (HPC), we tackle the high non-linearity of the problem to minimize, using global optimization methods which are easy to parallelize, in particular, evolution strategies. The first contribution adapt evolution strategies to the FWI setting where the cost function evaluation is the most expensive part. The second contribution is the parameterization of the regarded problem, by being able to represent the model, as faithfully as possible, while limiting the number of parameters needed, since each additional parameter is an additional dimension to explore. The last contribution is to propose a highly parallel evolution strategy adapted to the FWI setting. The initial results on the Salt Dome velocity model using low frequency range, show that great improvement can be done to automate the FWI

An outer approximation bi-level framework for mixed categorical structural optimization problems

In this paper, mixed categorical structural optimization problems are investigated. The aim is to minimize the weight of a truss structure with respect to cross-section areas, materials and cross-section type. The proposed methodology consists of using a bi-level decomposition involving two problems: master and slave. The master problem is formulated as a mixed integer linear problem where the linear constraints are incrementally augmented using outer approximations of the slave problem solution. The slave problem addresses the continuous variables of the optimization problem. The proposed methodology is tested on three different structural optimization test cases with increasing complexity. The comparison to state-of-the-art algorithms emphasizes the efficiency of the proposed methodology in terms of the optimum quality, computation cost, as well as its scalability with respect to the problem dimension. A challenging 120-bar dome truss optimization problem with 90 categorical choices per bar is also tested. The obtained results showed that our method is able to solve efficiently large scale mixed categorical structural optimization problems.Comment: Accepted for publication in Structural and Multidisciplinary Optimization, to appear 202

A Multiscale Parametrization for Refractivity Estimation in the Troposphere

This paper presents the idea of multiscale parametrization for tropospheric refractivity inversion using gradient-based optimization method. Our motivation is to improve the accuracy of inversion without the use of apriori information. We retrieve the details of the refractivity distribution progressively from large to smaller scales using hierarchical multiscale strategies in the admissible parameter space. The proposed formulation for multiscale adjoint tomography is validated and is confronted to a numerical test. This study shows that such strategies can potentially resolve complex ducting conditions which would otherwise fail a plain gradient-based inversion